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Abstract 

Time-delay mappings constructed using neural networks have proven successful in 
performing nonlinear system identification; however, because of their discrete nature, 
their use in bifurcation analysis of continuous-time systems is limited. This shortcoming 
can be avoided by embedding the neural networks in a training algorithm that mimics 
a numerical integrator. Both explicit and implicit integrators can be used. The former 
case is based on repeated evaluations of the network in a feedforward implementation; 
the latter relies on a recurrent network implementation. Here the algorithms and their 
implementation on parallel machines (SIMD and MIMD architectures) are discussed. 

1 Introduction 

Artificial Neural Networks (ANNs) have become a valuable tool for time series processing 
and nonlinear system identification tasks, particularly in those cases where models based 
on first principles are not quantitatively predictive, or simply not available. The ad hoc 
models obtained by using experimental measurements to train ANNs can lead to qualitative 
understanding of the long-term dynamic behavior of the system (its attractors) and their 
dependence on operating parameters. 

For deterministic systems, the future state can in general be obtained as a function 
of current and previous measurements of a single state variable (e.g. [10]). This idea has 
been successfully exploited to construct time-delay mappings based on highly parallelizable 
ANN implementations [3]. The ANN constructed typically consists of a linear input layer, 
one or more (nonlinear) hidden layers and a linear output layer. The current and delayed 
measurements of the state variable are the inputs to the ANN; additional input neurons 
can be used to incorporate the dependence on parameters. The output neuron(s) give the 
prediction of the state variable(s) at a future measurement time. 

The dicrete nature of mappings constructed this way prevents them from capturing 
certain qualitative features of the long term attractors and the bifurcation scenario of the 
continuous-time system they model (for a discussion see [8]). 

An alternative for identification of continuous-time systems, consisting of embedding the 
ANN in a numerical integrator scheme, has been developed in previous work [1, 7, 8]. Both 
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Fig. 1. ANN schematic for continuous-time identification, (a) A four-layered ANN based on 
a fourth order Runge-Kutta integrator, (h) ANN embedded in a simple implicit integrator. 

explicit and implicit integrators can be used; however, though the "explicit integrator" 
alternative can be readily implemented as a feedforward network, the use of implicit 
integrators requires recurrent network algorithms such as the one suggested in [6] (for 
another alternative involving recurrent networks see [4, 5]). 

2 Continuous-Time Modeling using Neural Networks 

Consider the set of autonomous ODEs [Y: state vector, X: parameters) 

(1) Y = f (y; X), YeW, XeW, f : 7^" X ^ 7^" 

We attempt to construct such a set of ODEs from discrete time data using state 
measurements only (i.e. without direct numerical evaluation of the time derivatives). The 
target value is the state of the system at some future time t+r. We attempt to approximate 
the unkown right hand side of the ODEs from the experimental knowledge of integrating XYas 
right hand side (for time r, with known initial conditions, the state at time t). We assume 
that the experimental data (the true trajectory of the underlying ODEs) are practically 
indistinguishable from the result of numerically integrating these ODEs using a simple 
numerical integration scheme. 

— * 

For a fourth order Runge-Kutta method, the result Y^+i of numerically integrating (1) 
for fixed values of the operating parameters X with initial conditions and a time step 
of h is given by: 

(2) = + ^(^1 + 2^2 + 2^ + ki), where : 

— * — * 

kr = M{Y^-X), k^ = M{Y^ + ^-X), k^ = M{Y^ + ^- X), K = M{Y^ + k^-X) 
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Fig. 2. (a) Periodic attractor of the Van der Pol oscillator for 7 = 1.0, 6 = 4.0 and ui = 1.0. 
The unstable steady state in the interior of the curve is marked +. (h) ANN-hased ■predictions for 
the attractors of the Van der Pol oscillator shown in (a). 



To approximate the unknown right hand side we use a four-layered ANN schematically 
described in Figure 1(a). Each ANN "box" (with sigmoidal activation functions in the 
hidden layers) evaluates i{Y;X), the right hand side of (1), given Y^. The above formulae 
can be used to derive the weight updating rules for training the overall network. Details of 
the implementation of this ANN-based procedure and some representative results can be 
found in [8]. 

An explicit integrator may face problems when the underlying system is "stiff . In this 
case, the identification procedure can be based on an implicit integrator scheme; a simple 
example is l^+i = -\- ^[f(l^;X) + f(l^_|_i;X). The resulting overall network is then 
recurrent [since some of its inputs are equal to its outputs, see figure 1(b)]. The prediction 
Y^+i of the state of the system at time t -\- h depends on itself. Training this network 
will require an on-line nonlinear solver, or an algortihm for recurrent networks such as the 
one suggested in [6]. Further details of the implementation of this algorithm can be found 
elsewhere [7]. 

In order to illustrate the capabilities of the methodology described above, we use data 
from the well-known Van der Pol oscillator: x = —7(2/^ — S)x — ojy,y = x. This system 
exhibits a stable periodic trajectory for 7 = 1.0, 6 = 4.0 and u) = 1.0. Figure 2(a) shows 
the periodic attractor obtained by numerical integration of the ODEs using a backward 
difference "stiff integrator with error control. The unstable steady state (with two positive 
eigenvalues) at (0,0) is marked + in figure 2(a). 

We use data converged on the periodic attractor to construct the training set for the 
ANN. A total of 240 points are used with "sampling" time 0.1 time units. The ANN "box" 
embedded in the implicit integrator consists of 2 linear input neurons, two hidden layers with 
5 neurons each and a two neuron (linear) output layer. Figure 2(b) shows the predictions 
given by long-term integration of the right hand side identified by the ANN. In spite of the 
large sampling time, the ANN correctly captures the character of the oscillations and even 
predicts the presence of the unstable steady state with good accuracy (both eigenvalues 
being positive). 

3 Parallel Implementation 

Simulation of recurrent networks is time consuming on a serial machine. Previous work 
[2] efficiently implemented feedforward networks on a parallel, SIMD, CM2 Connection 
Machine. The key to the efficiency was the "data parallel" nature of the algorithm [9], 
spreading the training examples over the processors of the parallel machine: each processor 
calculates the output of the net for the examples assigned to it. Since one typically has many 
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more examples than synaptic weights, this approach is more efficient on SIMD machines 
than parallelizing the neurons of a single network over many processors. For recurrent 
networks, the calculation of the output requires solving the following differential equation 
for the neuron activity Xi [6]: 

(3) X. = -aX, + + /,) 

3 

where, Wij are the synaptic weights, li are external currents, and a and /3 are fixed 
constants. The solution may be implemented on a SIMD machine if each processor performs 
the same number of steps of a discrete Euler integration. Alternatively, the calculation 
may be performed on a MIMD machine, such as the CM5, using any number of algorithms 
to find the fixed point of (3). The next step of the backpropagation algorithm requires 
propagation of the "error signal", Z^, through the network. This step requires the solution 
of a linear set of equations for Zi [6]: Lri'^r = •^i-, where Lri is a matrix defined in terms 
of the synaptic weights and neuron activities, and ]{ is defined in terms of the desired 
fixed points of (3). In the neural literature the solution, Z^, is defined in terms of the fixed 
point of the linear "network" equation: Zi = Lj-iZj. + J^. Euler integration of these 
ODEs may be efficiently performed on SIMD machines for a fixed number of time steps. 
One can alternatively employ a MIMD implementation to solve the above sets of linear 
equations. The generalization of our previous "data parallel" SIMD implementations of 
feedforward neural networks can be easily extended to recurrent networks. Utilization of 
MIMD machines, such as the CM5, can further increase computational efficiency. 
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